Why, in Deep Learning, Non-smooth Activation Function Works Better Than Smooth Ones

Since in the physical world, most dependencies are smooth (differentiable), traditionally, functions were used to approximate these dependencies. In particular, neural networks activation such as sigmoid function. However, successes of deep learning showed that many cases, non-smooth like $$\max (0,z)$$ work much better. this paper, we explain why approximating often better—even when approximated dependence is smooth.